Image pickup apparatus equipped with display section and method of controlling the same

ABSTRACT

An image pickup apparatus that is capable of reducing degradation of image quality of the whole display screen of a display section while improving display delay with respect to an observed area. In an image pickup apparatus, an image pickup section picks up an image of an object, and a development processor generates image data by developing input data acquired by the image pickup section. A first image generation unit generates a first image by performing image processing on part of the image data. A second image generation unit performs predetermined image processing on the image data to thereby generate a second image which is different in the number of steps of image processing from the above image processing. An image synthesis section generates a third image by synthesizing the first image with the second image, and a display device displays the third image.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image pickup apparatus equipped witha display section that displays an image to be photographed, and themethod of controlling the image pickup apparatus.

Description of the Related Art

A compact digital camera and a non-reflex camera (lens interchangeablecamera without a mirror box) are each equipped with a small-sizedelectronic viewfinder (EVF) for checking an object to be photographed,differently from a single-lens reflex camera. The camera of this typedoes not need to be equipped with a mirror box by employing the EVF, andhence it is possible to realize reduction of the size and weight of theapparatus body. On the other hand, a certain time period is required toperform image processing for displaying an image to be photographed onthe EVF. Therefore, there is a problem that a time difference between animage displayed on the EVF and an object image to be actuallyphotographed is generated, which makes it difficult to photograph theobject at a moment intended by the photographer.

Conventionally, most of users of compact digital cameras are notprofessional cameramen, and hence the above-mentioned display delay inthe EVF has not been considered as a serious problem.

However, in recent years, products for amateurs having high awareness ofphotographing techniques or professional cameramen, represented by ahigh-class compact camera and a mirrorless camera, have been increasedin number, and the problem of display delay cannot be ignored.

To solve the above-mentioned problem of display delay, there has beenproposed a technique for reading images from an image pickup device,starting with an object area observed by a user (observed area), andsequentially displaying the images on the EVF starting with an imageread from the observed area. Further, there have been proposed atechnique for changing the order of reading images and the order ofdisplaying images on the EVF, and further, a technique for omittingimage processing with respect to images other than the image read fromthe observed area (see e.g. Japanese Patent Laid-Open Publication No.2006-60496).

However, although the above-mentioned technique for reading an imagefrom the observed area first and displaying the image read from theobserved area first makes it possible to reduce the time of displaydelay with respect to the observed area, it requires a dedicated imagepickup device which can perform a special reading operation, and adedicated EVF which can display images in a special order adapted to thededicated image pickup device. Further, the technique for omitting imageprocessing with respect to images other than the image read from theobserved area has a problem that image quality of the whole EVF displayimage is degraded due to omission of image processing, and if theobserved area is in the vicinity of the center of the displayed image,distortion of the image due to the lens becomes noticeable in theperipheral part of the image.

SUMMARY OF THE INVENTION

The present invention provides an image pickup apparatus that is capableof reducing degradation of image quality of the whole display screen ofa display section while improving display delay with respect to anobserved area, and a method of controlling the image pickup apparatus.

In a first aspect of the invention, there is provided an image pickupapparatus comprising an image pickup unit configured to pick up an imageof an object, a development unit configured to generate image data bydeveloping input data acquired by said image pickup unit, a firstgeneration unit configured to generate a first image by performing imageprocessing on part of the image data, a second generation unitconfigured to perform predetermined image processing, which is larger inthe number of processing steps than the image processing performed bysaid first generation unit, on the image data to thereby generate asecond image, a synthesis unit configured to generate a third image bysynthesizing the first image with the second image, and a display unitconfigured to display the third image.

In a second aspect of the invention, there is provided a method ofcontrolling an image pickup apparatus comprising picking up an image ofan object, generating image data by developing input data acquired bysaid picking-up of the image, generating a first image by performingimage processing on part of the image data, performing predeterminedimage processing, which is larger in the number of processing steps thanthe image processing, on the image data to thereby generate a secondimage, generating a third image by synthesizing the first image with thesecond image, and displaying the third image.

According to the present invention, the first image generated by usingpart of image data, and the second image generated by performingpredetermined image processing on the image data, which is different inthe number of steps of image processing from image processing performedon the first image, are synthesized to thereby generate the third image,and the generated third image is displayed on the display unit. As aconsequence, it is possible to improve the display delay with respect tothe observed area by using part of the image data corresponding to theobserved area to generate the first image, and at the same time reducedegradation of image quality of the whole display screen of the displaysection by performing image processing on the other part of the imagedata than the part of the same corresponding to the observed area.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of an image pickup apparatusaccording to a first embodiment.

FIG. 2 is a diagram showing an example of pixel array of an image pickupdevice.

FIG. 3 is a flowchart of an image display process performed by the imagepickup apparatus according to the first embodiment.

FIGS. 4A to 4C are diagrams each showing an example of a display screenwhich is displayed on a display device of the image pickup apparatusaccording to the first embodiment.

FIG. 5 is a functional block diagram of an image pickup apparatusaccording to a second embodiment.

FIG. 6 is a flowchart of an image display process performed by the imagepickup apparatus according to the second embodiment.

FIG. 7 is a diagram showing an example of a display screen which isdisplayed on a display device of the image pickup apparatus according tothe second embodiment.

FIG. 8 is a functional block diagram of an image pickup apparatusaccording to a third embodiment.

FIG. 9 is a flowchart of an image display process performed by the imagepickup apparatus according to the third embodiment.

FIG. 10 is a diagram showing an example of a display screen which isdisplayed on a display device of the image pickup apparatus according tothe third embodiment.

FIG. 11 is a functional block diagram of an image pickup apparatusaccording to a fourth embodiment.

FIG. 12 is a flowchart of an image display process performed by theimage pickup apparatus according to the fourth embodiment.

DESCRIPTION OF THE EMBODIMENTS

The present invention will now be described in detail below withreference to the accompanying drawings showing embodiments thereof.

FIG. 1 is a functional block diagram of an image pickup apparatusaccording to a first embodiment. This image pickup apparatus is anapparatus configured to perform photographing by setting an area aroundthe center of a display screen as an observed area, and reduce displaydelay time for displaying an image of the observed area. The displaydelay time is a time period that elapses after an image of an object ispicked up by an image pickup section before the image is displayed on adisplay section.

Referring to FIG. 1, the image pickup apparatus, denoted by referencenumeral 201, is mainly comprised of a lens group 100, an image pickupsensor section 101, a first image generation unit 120, a second imagegeneration unit 130, an image synthesis section 110 that synthesizes afirst image and a second image, and a display device 111 that displays athird image. The first image generation unit 120 includes a developmentprocessor 102 and an image processor A 103. The second image generationunit 130 includes a storage section B 104 and an image processor B 105,a storage section C 106 and an image processor C 107, and a storagesection D 108 and an image processor D 109.

In this configuration, the lens group 100 adjusts an amount of incidentlight from an object as well as the focus, and forms a picked up imageon the image pickup sensor section 101. The image pickup sensor section101, which is formed by an image pickup device, such as a CCD or CMOSsensor, photoelectrically converts the image acquired from the lensgroup 100, further converts the image from analog to digital, andoutputs the digital signal. Pixels of the image pickup device each haveone of color filters of R (red), G (green), and B (blue), which arearranged e.g. in a mosaic pattern.

FIG. 2 is a diagram showing an example of pixel array of the imagepickup device. Referring to FIG. 2, the pixels of the image pickupdevice have a structure formed by regularly arraying sets of pixels in amosaic pattern in which each set is formed by four pixels, i.e. one redpixel 402, one blue pixel 401, and two green pixels 400. This pixelarray is generally referred to as the Bayer array.

An electric signal converted by the image pickup sensor section 101 to aBayer image signal is sent to the development processor 102 of the firstimage generation unit 120. The development processor 102 receives theBayer image signal, and performs RGB offset adjustment, gain adjustment,and gamma correction processing on the received image signal. That is,for example, the development processor 102 adjusts white balance byperforming offset adjustment and gain adjustment on the received Bayerimage signal. In gamma correction, by taking into account thecharacteristics of the image pickup sensor section 101 and the lensgroup 100, the image signal is processed using a gamma correction valuesuitable for generating a recording image desired by a user of the imagepickup apparatus. By changing the gamma correction value, it is alsopossible to generate a recording image on which texture and gradation ofa movie film are reproduced, and a recording image to be displayed on aTV monitor.

After performing gamma correction processing, the development processor102 generates an image having RGB image signals converted to a luminancesignal (Y) and color difference signals (Cb, Cr), and stores thegenerated image in the storage section B 104. The image processor A 103of the first image generation unit 120 generates a first image using animage of an observed area, which is part of the image data processed bythe development processor 102. In doing this, the image processor A 103determines the size of the first image as part of the image data and aposition of the first image in the whole image data according to theobserved area set by the user in advance.

The image processor B 105, the image processor C 107, and the imageprocessor D 109 of the second image generation unit 130 sequentiallyperform image processing on the image stored in the storage section B104. For example, the image processor B 105 performs processing for lensdistortion aberration correction, the image processor C 107 performsnoise elimination processing, and the image processor D 109 performscolor adjustment processing.

More specifically, the image processor B 105 performs e.g. processingfor lens distortion aberration correction on the image stored in thestorage section B 104, and stores an output image from the imageprocessor B 105 in the storage section C 106. Further, the imageprocessor C 107 performs e.g. noise elimination processing on the imagestored in the storage section C 106, and stores an output image from theimage processor C 107 in the storage section D 108. Further, the imageprocessor D 109 performs e.g. color adjustment processing on the imagestored in the storage section D 108, and sets an output image from theimage processor D 109 as a second image.

Assuming that an amount of delay between an output image from thedevelopment processor 102 and an output image from the image processor A103 is “delay 0” which is a delay amount reference, an amount of delayin the output image from the image processor B 105, which is caused byprocessing using the storage section B 104 and the image processor B105, is represented by “delay +1”, and an amount of delay in the outputimage from the image processor C 107, which is caused by processingusing the storage section C 106 and the image processor C 107, isrepresented by “delay +2”. Further, an amount of delay in the outputimage from the image processor C 107, which is caused by processingusing the storage section D 108 and the image processor D 109, isrepresented by “delay +3”. The unit of the amount of delay is e.g. atime period for processing one frame (frame time period). For example, adifference in delay between the first image and the second image is e.g.three frame time periods.

The image synthesis section 110 generates the third image bysynthesizing the first image output from the image processor A 103 andthe second image output from the image processor D 109. The displaydevice 111 is an EVF or a display, and displays the third imagesynthesized by the image synthesis section 110.

The processors of the image pickup apparatus 201 are each controlled bya CPU of a controller, not shown, to perform a predetermined operation,whereby the third image generated by synthesizing the first image andthe second image is displayed on the display device 111.

Next, an image display process performed by the image pickup apparatus201 shown in FIG. 1 will be described.

FIG. 3 is a flowchart of the image display process performed by theimage pickup apparatus according to the first embodiment. This imagedisplay process is performed by the CPU of the controller, not shown, ofthe image pickup apparatus 201 shown in FIG. 1, through execution of animage display process program stored in a ROM, also not shown.

Referring to FIG. 3, when the image display process is started, first,the CPU controls the lens group 100 and the image pickup sensor section101 to start photographing (step S101). Then, the CPU controls thedisplay device 111 to display the second image which is the output imagefrom the image processor D 109 on the display device 111 (step S102).The second image is an image generated by performing predeterminedprocessing on the image processed by the development processor 102 andstored in the storage section B 104, using the image processor B 105,the image processor C 107, and the image processor D 109.

Then, the CPU determines whether or not delay improved display has beenselected by a user (step S103). Selection of the delay improved displaymay be configured to be automatically set when the user presses ashutter button, or can be selectively set according to a user'sinstruction e.g. for a mode setting. If it is determined in the stepS103 that the delay improved display has been selected (YES to the stepS103), the CPU designates, for example, a central part of the displayscreen of the display device 111 as an observed area, according to asetting made by the user in advance (step S104).

FIGS. 4A to 4C are diagrams each showing an example of the displayscreen displayed on the display device of the image pickup apparatusaccording to the first embodiment, in which FIG. 4A shows a state of animage in which the observed area is designated by a broken-line frame,and FIG. 4B shows an image in a state after image substitution in whichthe frame of the observed area is explicitly indicated using solidlines. Further, FIG. 4C shows a third image in which an image of theobserved area (first image) is enlarged and then synthesized with thesecond image. In FIG. 4A, the central part of the screen is designatedas the observed area which is surrounded by the broke-line frame.

It is envisaged that the size of the observed area may be set by amethod of designating the number of pixels, a method of designating aratio of the observed area to the whole display screen, a method offollowing a user's instruction, and a method of making the size variabledepending on the size of an object to be observed. Further, the positionof the observed area is not limited to the central part of the screen,but for example, if there is no object to be observed in the centralpart of the screen, the observed area may be at a location other thanthe central part. The method of selecting the observed area and the sizeand position of the observed area are not particularly limited.

Referring back to FIG. 3, after the observed area is set in the centralpart of the image (step S104), the CPU generates the first image usingpart of the image corresponding to the observed area, based on theoutput value from the development processor 102 (step S105). Then, theCPU controls the image synthesis section 110 to generate the third imageby replacing the observed area of the second image with the first image(step S106). When replacing the image, the second image and the firstimage are different in delay time and the number of steps of imageprocessing, and hence misregistration or quality difference between theimages may be caused. The method of generating the third image is notlimited to a method of simply replacing an image, but, as shown in FIG.4B, may be a method of intentionally showing a frame of the observedarea after image replacement using solid lines to make the user aware ofthe frame, thereby reducing the sense of oddness. Further, the boundarybetween the first image and the second image can be made inconspicuousby blurring the boundary by filter processing, or gradually changing thesynthesis ratio. The observed areas in the first image and the secondimage are generally set to the same size and position in the displayscreen. However, besides displaying the observed area without changingan angle of view, the third image may be generated by synthesizing animage generated by enlarging the observed area of the first image, withthe second image, as shown in FIG. 4C, to thereby assist the user inchecking an image of the observed area. These methods of display may beadaptively selected according to a scene, or further, may be setaccording to a user's instruction. Note that details of the methods offilter processing and synthesis processing per se are known, and hencedescription thereof is omitted.

Referring back to FIG. 3, after the third image is generated, the CPUcontrols the display device 111 to display the third image on thedisplay screen of the display device 111 (step S107), followed byterminating the present process. Then, the CPU starts recording theimage of the object.

On the other hand, if it is determined in the step S103 that the delayimproved display has not been selected (NO to the step S103), the CPUreturns to the step S102, and waits for the user to select the delayimproved display while displaying the second image.

According to the image display process in FIG. 3, the processingoperations performed by the image processor B 105, the image processor C107, and the image processor D 109 are omitted with respect to the firstimage as the image of the observed area, and hence it is possible toimprove delay conventionally caused before the observed area isdisplayed on the display device. Further, the first image as the imageof the observed area is set in a central area of the display screen, andhence it is possible to prevent distortion in the peripheral part of theimage, which is liable to occur in the third image generated bysynthesizing the first image with the second image. Note that basicimage processing including offset adjustment, gain adjustment, and gammacorrection is performed by the development processor 102, and hence itis possible to prevent image quality of the whole display screen frombeing extremely degraded.

In the present embodiment, as the storage section B 104, the storagesection C 106, and the storage section D 108, a memory, such as a RAM,is used. These storage sections may be formed by a single storagesection which is used through address control, or may be formed by aplurality of respective storage sections.

In the present embodiment, it is desirable that synchronization of thedriving period of the image pickup device, the driving period of eachstorage section, and the driving period of the EVF can be controlled.For example, an amount of delay between synchronization signals for theimage pickup device and the EVF may be measured, and with reference tothe driving period of the image pickup device, it is possible to controlthe amount of delay of each of synchronization signal for the respectivedriving periods of the storage sections and the driving period of theEVF, such that the amount of delay is within a predetermined range. Thedriving period of the image pickup device and that of the EVF can beeasily controlled if the image pickup device and the EVF are driven atthe same period. However, the driving period of the image pickup deviceand that of the EVF are not limited to the same period.

In the present embodiment, it is preferable that the image processor A103 performs at least one of processing for lens distortion aberrationcorrection, noise elimination processing, and color adjustmentprocessing, on part of the image data generated by the developmentprocessor 102. This makes it possible to improve the image quality ofthe first image which is synthesized with the second image, and therebymake the image quality of the first image close to that of the secondimage. The number of steps of image processing performed by the imageprocessor A 103 on the above-mentioned part of the image data isdetermined according to e.g. the display delay time allowed by the user.

In the present embodiment, the development processor 102 performs atleast one of RGB offset adjustment, gain adjustment, and gammacorrection processing, on the Bayer image signal as the input data. Arange of processing to be performed is determined e.g. based on thedisplay delay time allowed by the user.

In the present invention, in synthesis processing for synthesizing thefirst image with the second image, it is preferable to perform filterprocessing or the like on a boundary between the second image and thefirst image, and thereby make the boundary inconspicuous.

In the present embodiment, it is preferable that when synthesizing thefirst image with the second image, the image synthesis section 110synthesizes the images by taking into account hand shake correction anda motion of the camera (panning). This makes it possible to reduce adifference in motion at the boundary between the second image and thefirst image.

Next, a description will be given of a second embodiment of the presentinvention with reference to the drawings.

FIG. 5 is a functional block diagram of an image pickup apparatusaccording to the second embodiment. This image pickup apparatus is anapparatus which is configured to perform photographing by determining anobserved area by an object recognition function, and is capable ofcoping with a case where the observed area desired by a user is not in acentral area of the screen.

Referring to FIG. 5, the image pickup apparatus, denoted by referencenumeral 202, has the same basic configuration as that of theabove-described image pickup apparatus 201 shown in FIG. 1. Therefore,the same components as those of the image pickup apparatus 201 aredenoted by the same reference numerals, and description thereof isomitted. The following description will be given mainly of differentpoints of the image pickup apparatus 202 according to the presentembodiment from the image pickup apparatus 201 according to the firstembodiment.

The image pickup apparatus 202 differs from the image pickup apparatus201 shown in FIG. 1 in that a recognition section 112 is providedbetween the development processor 102 and the image processor A 103 ofthe first image generation unit 120. The recognition section 112 is ablock that recognizes a face of a person as an object, identifies aperson, or recognizes a physical object. Although the recognitionsection 112 is formed as a block independent of the image processor A103 for the convenience of explanation, the image processor A 103 towhich the function of the recognition section 112 is added can also beused.

FIG. 6 is a flowchart of an image display process performed by the imagepickup apparatus according to the second embodiment. This image displayprocess is performed by the CPU of the controller, not shown, of theimage pickup apparatus 202, through execution of an image displayprocess program stored in the ROM, also not shown.

The following description will be given mainly of different points ofthe image display process performed by the image pickup apparatus 202shown in FIG. 5 from the image display process (FIG. 3) performed by theimage pickup apparatus 201 according to the first embodiment.

Referring to FIG. 6, when the image display process is started,similarly to the above-described image display process performed by theimage pickup apparatus 201 according to the first embodiment, the CPUstarts photographing, controls the display device 111 to display thesecond image on the display device 111, and determines whether or notthe delay improved display has been selected (steps S201 to S203).

If it is determined in the step S203 that the delay improved display hasbeen selected (YES to the step S203), the CPU sets a condition of theobserved area (step S204). In the present embodiment, it is assumed thatthe condition is that a “face” of a person or a “bouquet of flowers” isrecognized in the observed area. The condition of the observed area maybe designated from the outside after a user confirms an image on thedisplay device 111 or may be designated by registering information on aphysical object in advance.

After the condition of the observed area is set (step S204), the CPUcauses the image recognition operation to be performed (step S205). Thatis, the CPU controls the recognition section 112 to recognize an imagewhich satisfies the condition set for determining the observed area,such as a face and a bouquet, on the display screen of the displaydevice 111.

Then, the CPU determines whether or not an image which satisfies thecondition of the observed area has been recognized (step S206). If it isdetermined in the step S206 that the recognition section 112 hasrecognized the image (face or bouquet) which satisfies the condition ofthe observed area (YES to the step S206), the CPU sets the recognizedarea as the observed area (step S207), as shown in FIG. 7.

FIG. 7 is a diagram showing an example of the display screen displayedon the display device of the image pickup apparatus according to thesecond embodiment. FIG. 7 shows a case where a plurality of observedareas are set, and more specifically, in the illustrated example, a faceof a person and a bouquet, surrounded by broken lines, are set as theobserved areas.

After the observed areas are set (step S207), the CPU cuts out aplurality of parts of images corresponding to the observed areas fromoutput values from the development processor 102 to thereby generatefirst images (step S208). Then, the CPU replaces the images of theobserved areas of the second image with the first images to therebygenerate the third image, similarly to the first embodiment (FIG. 3)(step S209). In doing this, the CPU replaces the images of the twoobserved areas of the second image with the two first imagescorresponding thereto. After generating the third image (step S209), theCPU controls the display device 111 to display the third image on thedisplay device 111 (step S210), followed by terminating the presentprocess. Then, the CPU starts recording the image.

On the other hand, if it is determined in the step S203 that the delayimproved display has not been selected (NO to the step S203), the CPUreturns to the step S202, and waits until the delay improved display isselected. Further, if it is determined in the step S206 that an imagesatisfying the condition of the observed area has not been recognized(NO to the step S206), the CPU returns to the step S205, and continuesthe image recognition operation.

According to the image display process in FIG. 6, similarly to the firstembodiment, the processing operations performed by the image processor B105, the image processor C 107, and the image processor D 109 areomitted with respect to the first images corresponding to the observedareas, and hence it is possible to improve the delay conventionallycaused before the first image is displayed on the display device.Further, since the recognition section 112 for recognizing the observedarea is provided, it is possible to quickly recognize the first image(s)associated with the observed area(s), and display the third imageobtained by synthesizing the first image(s) and the second image.Therefore, this also makes it possible to improve display delay.

Further, basic image processing including offset adjustment, gainadjustment, and gamma correction is performed by the developmentprocessor 102, and hence it is possible to prevent the image quality ofthe whole display screen from being very much degraded.

In the present embodiment, the number and size of the observed areas arenot particularly limited, but can be changed as desired by a user on anas-needed basis.

Next, a description will be given of a third embodiment of the presentinvention with reference to drawings.

FIG. 8 is a functional block diagram of an image pickup apparatusaccording to the third embodiment. This image pickup apparatus isconfigured to change details of processing performed when generating thefirst image, depending on the size of the observed area.

Referring to FIG. 8, the image pickup apparatus, denoted by referencenumeral 203, has the same basic configuration as that of theabove-described image pickup apparatus 201 shown in FIG. 1. Therefore,the same components as those of the image pickup apparatus 201 aredenoted by the same reference numerals, and description thereof isomitted. The following description will be given mainly of differentpoints of the image pickup apparatus 203 according to the presentembodiment from the image pickup apparatus 201 according to the firstembodiment.

The image pickup apparatus 203 differs from the image pickup apparatus201 shown in FIG. 1 in that an image processor E 113 and a first imageselection section 114 are provided in place of the image processor A 103of the first image generation unit 120, appearing in FIG. 1. The imageprocessor E 113 performs a plurality of different image processingoperations on image data output from the development processor 102 withrespect to the observed area to thereby generate a plurality of images,and outputs the generated images as images to be each used as the firstimage. Examples of the plurality of processing operations include thesame processing operations as those performed by the image processor A103, the image processor B 105, the image processor C 107, and the imageprocessor D 109, respectively, and processing operations which arepartially simplified versions of the above-mentioned processingoperations. The image processor E 113 outputs a plurality of images asimages to be each used as the first image, for example, an image 1, animage 2, and an image 3, to the first image selection section 114,according to different details of the processing.

The image 2 is an image subjected to higher-level processing than theimage 1, and the image 3 is an image subjected to even higher-levelprocessing than the image 2. The first image selection section 114determines the first image by selecting one image out of the pluralityof images as the images to be each used as the first image, which areoutput from the image processor E 113. In this case, the first imageselection section 114 selects one image using the size of the observedarea selected by the user as a reference. The reference for selectionwill be described hereinafter.

FIG. 9 is a flowchart of an image display process performed by the imagepickup apparatus according to the third embodiment. This image displayprocess is performed by the CPU of the controller, not shown, of theimage pickup apparatus 203, through execution of an image displayprocess program stored in the ROM, also not shown.

The following description will be given mainly of different points ofthe image display process performed by the image pickup apparatus 203shown in FIG. 8 from the image display process (FIG. 3) performed by theimage pickup apparatus 201 according to the first embodiment.

Referring to FIG. 9, when the image display process is started,similarly to the above-described image display process performed by theimage pickup apparatus according to the first embodiment, the CPU startsphotographing, controls the display device 111 to display the secondimage on the display device 111, and determines whether or not the delayimproved display has been selected (steps S301 to S303).

If it is determined in the step S303 that the delay improved display hasbeen selected (YES to the step S303), the CPU sets a range of theobserved area which is designated e.g. by a user (step S304). FIG. 10shows a display screen displayed on the display device of the imagepickup apparatus according to the third embodiment, for prompting theuser to designate a range of the observed area. Referring to FIG. 10, arange A surrounded by dashed single-dotted lines and a range Bsurrounded by dashed double-dotted lines are shown on the displayscreen. For example, the user designates an observed area having adesired size by selecting one of the areas on the screen shown in FIG.10. Besides designating the range of the observed area according touser's selection as described above, the range of the observed area maybe designated by a method of registering information on a physicalobject as a setting in advance and designating a range whichaccommodates the physical object on which the information is registered.

For example, after the range of the observed area is set based on theuser's designation (step S304), the CPU determines whether or not theobserved area which has been recognized is larger than the range A inFIG. 10 (step S305). If it is determined in the step S305 that theobserved area is larger than the range A (YES to the step S305), the CPUselects the image 1 processed by the first image generation unit 120appearing in FIG. 8, as the first image, and sends the image 1 to theimage synthesis section 110. Then, the CPU controls the image synthesissection 110 to generate the third image by synthesizing the image 1 withthe second image (step S306), and controls the display device 111 todisplay the third image on the display device 111 (step S310), followedby terminating the present process. Note that the method of generatingthe third image is the same as that used in the first embodiment.

On the other hand, if it is determined in the step S305 that theobserved area is not larger than the range A (NO to the step S305), theCPU determines whether or not the observed area is larger than the rangeB (step S307). If it is determined in the step S307 that the observedarea is larger than the range B (YES to the step S307), the CPU selectsthe image 2 generated by the first image generation unit 120 appearingin FIG. 8, as the first image, and sends the image 2 to the imagesynthesis section 110. Then, the CPU controls the image synthesissection 110 to generate the third image by synthesizing the image 2 withthe second image (step S308), and then proceeds to the step S310.

Further, if it is determined in the step S307 that the observed area isnot larger than the range B (NO to the step S307), the CPU selects theimage 3 generated by the first image generation unit 120 appearing inFIG. 8, as the first image, and sends the image 3 to the image synthesissection 110. Then, the CPU controls the image synthesis section 110 togenerate the third image by synthesizing the image 3 with the secondimage (step S309), and then proceeds to the step S310.

On the other hand, if it is determined in the step S303 that the delayimproved display has not been selected (NO to the step S303), the CPUreturns to the step S302, and waits until the delay improved display isselected.

According to the image display process in FIG. 9, it is possible tochange details of processing performed when generating the first imageaccording to a size of the observed area which is prepared in pluralitywith respective different sizes. For example, as processing operationsto be performed in a case where the observed area is the smallest, it ispossible to select from a processing operation performed by the imageprocessor A 103 appearing in FIG. 1, and all or most of processingoperations performed by the image processor B 105, the image processor C107, and the image processor D 109, each appearing in FIG. 8. When thescreen is small, it does not take much time to complete processing andhence it is possible to perform complicated processing. This makes itpossible to improve the image quality of the observed area tosubstantially the same level as the second image, and hence it ispossible to prevent the image quality of the whole display image frombeing degraded.

In the present embodiment, as the observed area is smaller, the imageprocessor E 113 makes higher in level the details of processingperformed with respect to the observed area. For example, the smallestimage 3 is an image generated by performing processing on part of theimage processed by the development processor 102, using the imageprocessor B 105, the image processor C 107, and the image processor D109. Further, the image 2 larger than the image 3 is an image generatedby performing processing on part of the image processed by thedevelopment processor 102, using the image processor B 105 and the imageprocessor C 107. Further, the image 1 larger than the image 2 is animage generated by performing processing on part of the image processedby the development processor 102, only using the image processor B 105.

In the present embodiment, processing performed by the image processor B105 is e.g. processing for lens distortion aberration correction.Further, processing performed by the image processor C 107 is e.g. noiseelimination processing, and processing performed by the image processorD 109 is e.g. color adjustment processing.

In the present embodiment, details of the processing performed to obtainthe image 1, the image 2, and the image 3 are preferably determinedaccording to processing time which can be assigned to the imageprocessor E 113.

Although in the present embodiment, the range A and the range B aredescribed as examples of the range for reference for use in setting anobserved area, for the convenience of explanation, the number of theranges is not particularly limited. Therefore, the number of the rangesmay be increased according to the types of image processing performed bythe image processor E 113 or time periods required to perform imageprocessing performed by the same.

Further, in the present embodiment, the number of outputs from the imageprocessor E 113 is not particularly limited, but it is only required tobe one or more. Note that the time required to perform processing by theimage processor E 113 is different depending on details of theprocessing, and hence there is a case where the processing is notcompleted within a desired time period, depending on the size of theobserved area. Therefore, the first image selection section 114 iscontrolled not to select an image on which processing has not beencompleted even if the image is the one output from the image processor E113.

Next, a description will be given of a fourth embodiment of the presentinvention with reference to the drawings.

FIG. 11 is a functional block diagram of an image pickup apparatusaccording to the fourth embodiment. This image pickup apparatus isconfigured to change details of processing performed when generating thefirst image according to the length of display delay time allowed by auser.

A time period allowable as the display delay time is different dependingon each user. If the allowable time period is long, image processing forimproving display image quality can be performed accordingly. On theother hand, if the allowable time is short, it is required to make theimage processing time as short as possible. In this case, the time spentfor performing image processing is short and hence image quality isdegraded accordingly.

Referring to FIG. 11, the image pickup apparatus, denoted by referencenumeral 204, has the same basic configuration as that of theabove-described image pickup apparatus 201 shown in FIG. 1. Therefore,the same components as those of the image pickup apparatus 201 aredenoted by the same reference numerals, and description thereof isomitted. The following description will be given mainly of differentpoints of the image pickup apparatus 204 according to the presentembodiment from the image pickup apparatus 201 according to the firstembodiment.

The image pickup apparatus 204 shown in FIG. 11 differs from the imagepickup apparatus 201 shown in FIG. 1 in that a first image determinationsection 115 is added. The first image determination section 115determines the first image from images output from the image processor A103 as images to be each used as the first image, an image B output fromthe image processor B 105, and an image C output from the imageprocessor C 107, and sends the determined first image to the imagesynthesis section 110. Note that the image processor A 103 generates oneor more images as images to be each used as the first image. Further,the image B and the image C are output from the image processor B 105and the image processor C 107, respectively, and are cut out as imageswhich have the same size as the first image and include the sameobserved area as the first image. The image B and the image C correspondto the above-mentioned images to be each used as the first image, and inother words, the other images to be each used as the first image.

FIG. 12 is a flowchart of an image display process performed by theimage pickup apparatus according to the fourth embodiment. This imagedisplay process is performed by the CPU of the controller, not shown, ofthe image pickup apparatus 204, through execution of an image displayprocess program stored in the ROM, also not shown.

The following description will be given mainly of different points ofthe image display process performed by the image pickup apparatus 204shown in FIG. 11 from the image display process (FIG. 3) performed bythe image pickup apparatus 201 according to the first embodiment.

Referring to FIG. 12, when the image display process is started,similarly to the above-described image display process performed by theimage pickup apparatus according to the first embodiment, the CPU startsphotographing, and controls the display device 111 to display the secondimage on the display device 111, and determines whether or not the delayimproved display has been selected (steps S401 to S403).

If it is determined in the step S403 that the delay improved display hasbeen selected (YES to the step S403), the CPU confirms the display delaytime selected by the user (step S404). The user sets the display delaytime, which is allowable, in advance or in response to a request fromthe CPU. The display delay time is set to D0=3 ms, D1=5 ms, or D2=10 ms,for example.

After confirming the setting of the display delay time (step S404), theCPU determines whether or not D0 has been selected as the display delaytime (step S405). If it is determined in the step S405 that D0 has beenselected (YES to the step S405), the CPU controls the first imagedetermination section 115 to select one of the images to be each used asthe first image as the first image in FIG. 11. Then, the CPU controlsthe image synthesis section 110 to generate the third image bysynthesizing the selected first image with the second image (step S406),and controls the display device 111 to display the third image on thedisplay device 111 (step S410), followed by terminating the presentprocess.

On the other hand, if it is determined in the step S405 that D0 has notbeen selected by the user (NO to the step S405), the CPU determineswhether or not the display delay time D1 has been selected by the user(step S407). If it is determined in the step S407 that D1 has beenselected (YES to the step S407), the CPU controls the first imagedetermination section 115 to select the image B in FIG. 11 as the firstimage, and controls the image synthesis section 110 to generate thethird image by synthesizing the selected first image with the secondimage (step S408). Then the process proceeds to the step S410.

Further, if it is determined in the step S407 that the display delaytime D1 has not been selected (NO to the step S407), the CPU controlsthe first image determination section 115 to select the image C in FIG.11 as the first image, and controls the image synthesis section 110 togenerate the third image by synthesizing the selected first image withthe second image (step S409). Then the process proceeds to the stepS410. Note that the method of generating the third image is the same asthat used in the first embodiment.

On the other hand, if it is determined in the step S403 that the delayimproved display has not been selected (NO to the step S403), the CPUreturns to the step S402, and waits until the delay improved display isselected.

According to the process in FIG. 12, when the display delay time allowedby the user is D0 (3 ms), the third image is formed by selecting, as thefirst image, the image to be used as the first image, which is generatedby performing only image processing using the image processor A 103.Alternatively, when the display delay time allowed by the user is D1 (5ms), the third image is formed by selecting, as the first image, theimage B which is generated by performing image processing using theimage processor B 105. Further alternatively, when the display delaytime allowed by the user is D2 (10 ms), the third image is formed byselecting, as the first image, the image C which is generated byperforming image processing using the image processor B 105 and theimage processor C 107. That is, an image formed by performing imageprocessing according to the display delay time allowed by the user isselected as the first image to be synthesized with the second image,whereby it is possible to preferably reduce degradation of image qualityof the whole display screen within an allowable range of display delay.

Although in the above-described embodiments, the three image processorsB 105, C 107, and D 109 have been described as an example of the imageprocessor for generating the second image, the image processor forgenerating the second image is not particularly limited to these. Thatis, any other image processor for generating the second image can beemployed insofar as it can execute at least one or more image processingoperations.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2014-204734 filed Oct. 3, 2014, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image pickup apparatus comprising a memorythat stores a program, and at least one processor that executes theprogram to cause the image pickup apparatus to function as: an imagepickup unit configured to pick up an image of an object; a developmentunit configured to generate image data by developing input data acquiredby said image pickup unit; a first generation unit configured togenerate a first image by performing image processing on part of theimage data as an image of an observed area; a second generation unitconfigured to perform predetermined image processing, which has agreater number of processing steps than the image processing performedby said first generation unit, on the image data to thereby generate asecond image; a synthesis unit configured to generate a third image bysynthesizing the first image with the second image; and a display unitconfigured to display the third image.
 2. The image pickup apparatusaccording to claim 1, wherein the part of the image data is image datacorresponding to the observed area set in a display screen of saiddisplay unit in advance.
 3. The image pickup apparatus according toclaim 2, wherein said first generation unit determines a size of thefirst image and a position of the first image in the whole image dataaccording to the observed area.
 4. The image pickup apparatus accordingto claim 1, wherein the image processing performed by said firstgeneration unit includes at least one of processing for lens distortionaberration correction, noise elimination processing, and coloradjustment processing.
 5. The image pickup apparatus according to claim1, wherein said first generation unit determines a number of processingsteps of the image processing performed on the part of the image dataaccording to an allowable display delay time period.
 6. The image pickupapparatus according to claim 1, wherein said first generation unitgenerates a plurality of first images by performing respective imageprocessings which are different in processing details, and selects oneimage from the plurality of generated first images according to a sizeof a set observed area, and wherein said synthesis unit synthesizes theselected first image with the second image.
 7. The image pickupapparatus according to claim 6, wherein, when the size of the setobserved area is a relatively small size, said first generation unitselects an image formed by image processing at a higher level of detail.8. The image pickup apparatus according to claim 6, wherein said firstgeneration unit generates the plurality of first images by performingthe respective image processings which are different in the number ofprocessing steps, on the part of the image data.
 9. The image pickupapparatus according to claim 1, wherein, in a case in which theplurality of observed areas are set, said first generation unitgenerates the first image using a plurality of parts of image data, andsaid synthesis unit forms the third image by synthesizing the firstimage formed by the plurality of parts, generated by said firstgeneration unit, with the second image.
 10. The image pickup apparatusaccording to claim 1, wherein said synthesis unit enlarges the firstimage, and then synthesizes the enlarged first image with the secondimage to thereby form the third image.
 11. The image pickup apparatusaccording to claim 1, wherein said synthesis unit forms the third imagein a manner gradually changing a synthesis ratio at a boundary betweenthe first image and the second image.
 12. The image pickup apparatusaccording to claim 1, wherein said synthesis unit forms the third imageby blurring a boundary between the first image and the second image byperforming filter processing.
 13. The image pickup apparatus accordingto claim 1, wherein said synthesis unit forms the third image byindicating a boundary between the first image and the second image usingframe lines.
 14. A method of controlling an image pickup apparatus, themethod comprising: picking up an image of an object; generating imagedata by developing input data acquired by said picking-up of the image;generating a first image by performing image processing on part of theimage data as an image of an observed area; performing predeterminedimage processing, which has a greater number of processing steps thanthe image processing, on the image data to thereby generate a secondimage; generating a third image by synthesizing the first image with thesecond image; and displaying the third image.